Information processing apparatus, information processing method, and program

ABSTRACT

An information processing apparatus ( 100 ) includes a collation unit ( 102 ) that collates first feature information extracted from a person included in a first image ( 20 ) with first registered feature information stored in a storage unit ( 110 ), and a registration unit ( 104 ) that stores, in a case where the first feature information is unable to be extracted from the person or a collation result in the collation unit ( 102 ) indicates a mismatch, second feature information extracted from the person in the storage unit ( 110 ) as second registered feature information, in which the collation unit ( 102 ) collates second feature information extracted from a person included in a second image ( 22 ) with the second registered feature information stored in the storage unit ( 110 ), and thus specifies the person corresponding to the second registered feature information in the second image ( 22 ).

TECHNICAL FIELD

This disclosure relates to an information processing system, aninformation processing apparatus, an information processing method, anda program, and particularly to an information processing system, aninformation processing apparatus, an information processing method, anda program, capable of performing an image recognition process.

BACKGROUND ART

Patent Document 1 discloses an example of a walk-through type face imageauthentication apparatus. The face image authentication apparatusdisclosed in Patent Document 1 specifies an authentication candidateperson from a plurality of faces (appearing persons) included in aplurality of facial region images captured at different time points, anddetermines an authentication threshold value by using an indexindicating a height of a similarity degree between the authenticationcandidate person and a face image. The index used here is (a first placehit ratio which is a ratio of a frame in which the similarity degree ishighest). As the first place hit ratio becomes higher, theauthentication threshold value is reduced. Even in a case where there isa frame in which the similarity degree with respect to another person istemporarily high due to a change of an expression or a direction of aface, when the first place hit ratio is low, the authenticationthreshold value is not reduced.

A person recognition apparatus disclosed in Patent Document 2 detects aface of a passerby from each of a first image captured by a first cameraand a second image captured by a second camera. Here, the first camerais provided in a state of being easily recognized by a passerby, and thesecond camera is provided in a state of being hardly recognized by apasserby. The passerby is classified on the basis of detection resultsin both of the images, an authentication threshold value used for a facecollation process is adjusted on the basis of a classification result,and an output process content based on an authentication result isdetermined.

A person judgment apparatus and a person retrieval tracking apparatusdisclosed in Patent Document 3 detect a walking state of a person fromtemporally-distant frames or person image sequences obtained bydifferent cameras, and judges whether or not persons included indifferent image sequences are the same person on the basis of thewalking state. Consequently, a person can be tracked. Patent Document 4discloses a technique in which a stride, a height, a weight, or awalking state (a stride or a pace) of an authentication target person isdetected, and personal authentication is performed through collationwith authentication information. Patent Document 5 discloses a gatemanagement system which performs an authentication process on the basisof a gait (a stride, a posture, a way to swing arms, or the like) of aperson passing through a gate, and controls locking and unlocking of thegate.

RELATED DOCUMENT Patent Document

[Patent Document 1] Japanese Patent Application Publication No.2013-101551

[Patent Document 2] Japanese Patent Application Publication No.2008-108243

[Patent Document 3] International Publication No. WO2006/013765

[Patent Document 4] Japanese Patent Application Publication No.2009-104526

[Patent Document 5] Japanese Patent Application Publication No.2018-77552

SUMMARY OF THE INVENTION Technical Problem

The present inventor has examined a new technique for tracking a personby using image processing. In other words, an object of this disclosureis to provide a new technique for tracking a person by using imageprocessing.

Solution to Problem

In each aspect of this disclosure, the following configuration isemployed to solve the above-described problem.

A first aspect relates to an information processing apparatus.

A first information processing apparatus related to the first aspectincludes a collation unit that collates first feature informationextracted from a person included in a first image with first registeredfeature information stored in a storage unit; and a registration unitthat, in a case where the first feature information is unable to beextracted from the person or a collation result in the collation unitindicates a mismatch, stores second feature information extracted fromthe person in the storage unit as second registered feature information,in which the collation unit collates second feature informationextracted from a person included in a second image with the secondregistered feature information stored in the storage unit, and thusspecifies the person corresponding to the second registered featureinformation in the second image.

A second information processing apparatus related to the first aspectincludes a collation unit that collates first feature informationextracted from a person included in a first image with first registeredfeature information stored in a storage unit; and a registration unitthat stores second feature information extracted from the person in thestorage unit as second registered feature information, in which thecollation unit collates second feature information extracted from aperson included in a second image with the second registered featureinformation stored in the storage unit, and thus specifies the personcorresponding to the second registered feature information in the secondimage, and in which, in a case where the first feature information isunable to be extracted from the person included in the first image, or acollation result between the first feature information extracted fromthe person included in the first image and the first registered featureinformation indicates a mismatch, the collation unit collates firstfeature information extracted from the person specified in the secondimage with the first registered feature information stored in thestorage unit.

A second aspect relates to an information processing method executed byat least one computer.

A first information processing method executed by an informationprocessing apparatus, related to the second aspect, includes: collatingfirst feature information extracted from a person included in a firstimage with first registered feature information stored in a storageunit; storing, in a case where the first feature information is unableto be extracted from the person or a collation result in the collationunit indicates a mismatch, second feature information extracted from theperson in the storage unit as second registered feature information; andcollating second feature information extracted from a person included ina second image with the second registered feature information stored inthe storage unit, and thus specifying the person corresponding to thesecond registered feature information in the second image.

A second information processing method executed by an informationprocessing apparatus, related to the second aspect, includes: collatingfirst feature information extracted from a person included in a firstimage with first registered feature information stored in a storageunit; storing second feature information extracted from the person inthe storage unit as second registered feature information; collatingsecond feature information extracted from a person included in a secondimage with the second registered feature information stored in thestorage unit, and thus specifying a person corresponding to the secondregistered feature information in the second image; and collating, in acase where the first feature information is unable to be extracted fromthe person included in the first image, or a collation result betweenthe first feature information extracted from the person included in thefirst image and the first registered feature information indicates amismatch, the first feature information extracted from the personspecified in the second image with the first registered featureinformation stored in the storage unit.

It should be noted that other aspects of this disclosure may relate to aprogram causing at least one computer to execute the method of thesecond aspect described above, and may relate to a computer readablestorage medium storing such a program. The storage medium includes anon-transitory medium.

The computer program includes computer program codes causing a computerto execute the information processing method on the informationprocessing apparatus when the program is executed by the computer.

It should be noted that any combination of the above-describedconstituent elements, and expressional conversion of this disclosureamong a method, an apparatus, a system, a storage medium, a computerprogram, and the like are also effective as an aspect of thisdisclosure.

Various constituent elements of this disclosure are not necessarilyrequired to be individually independent elements. For example, aplurality of constituent elements may be configured as a single member,a single constituent element may be configured with a plurality ofmembers, any constituent element may be a part of another constituentelement, and a part of any constituent element may overlap a part ofanother constituent element.

A plurality of procedures are sequentially described in the method andthe computer program of this disclosure, but the order of descriptiondoes not limit an order of executing the plurality of procedures. Thus,in a case where the method and the computer program of this disclosureare executed, the order of the plurality of procedures may be changedwithin the scope without contradiction to contents thereof.

The plurality of procedures of the method and the computer program ofthis disclosure are not limited to being individually executed atdifferent timings. Thus, another procedure may occur during execution ofany procedure, and an execution timing of any procedure may partially orentirely overlap an execution timing of another procedure.

Advantageous Effects of Invention

According to the respective aspects, it is possible to provide a newtechnique for detecting a person to whom attention is required to bepaid by using image processing.

BRIEF DESCRIPTION OF THE DRAWINGS

The above-described object, and other objects, features, and advantageswill become apparent throughout preferable example embodiments describedbelow and the accompanying drawings.

FIG. 1 is a functional block diagram illustrating a logicalconfiguration of an information processing apparatus according to afirst example embodiment of this disclosure.

FIG. 2 is a diagram illustrating an example of a data structure of asecond storage unit of a storage unit in FIG. 1.

FIG. 3 is a flowchart illustrating an example of an operation of theinformation processing apparatus in FIG. 1.

FIG. 4 is a diagram illustrating an example of a data structure of asecond storage unit of a storage unit of an information processingapparatus according to a second example embodiment of this disclosure.

FIG. 5 is a flowchart illustrating examples of procedures of step S125and the subsequent steps in FIG. 3 in an operation of the informationprocessing apparatus according to the second example embodiment of thisdisclosure.

FIG. 6 is a flowchart illustrating examples of procedures of step S135and the subsequent steps in FIG. 5 in an operation of the informationprocessing apparatus according to the second example embodiment of thisdisclosure.

FIG. 7 is a functional block diagram illustrating a logicalconfiguration of an information processing apparatus according to athird example embodiment of this disclosure.

FIG. 8 is a diagram for explaining a timing of information output in anoutput unit of the information processing apparatus in FIG. 7.

FIG. 9 is a diagram illustrating an example of a data structure of asecond storage unit of a storage unit in FIG. 7.

FIG. 10 is a flowchart illustrating an example of an operation of aninformation processing apparatus according to a fourth exampleembodiment of this disclosure.

FIG. 11 is a flowchart illustrating an example of an operation of aninformation processing apparatus according to a fifth example embodimentof this disclosure.

FIG. 12 is a flowchart illustrating an example of an operation of aninformation processing apparatus according to a sixth example embodimentof this disclosure.

FIG. 13 is a diagram illustrating an example of a computer implementingthe information processing apparatus of each example embodiment.

FIG. 14 is a diagram illustrating examples of display screens of imagesobtained by a camera imaging a predetermined area.

FIG. 15 is a diagram illustrating a change in data registered in asecond storage unit of a storage unit of an Example.

FIG. 16 is a diagram illustrating examples of other data structures ofthe second storage unit of the storage unit of the Example.

FIG. 17 is a diagram schematically illustrating a situation of a placewhere four cameras are provided along a certain passage.

FIG. 18 is a diagram illustrating a change in data registered in thesecond storage unit of the storage unit of the Example.

DESCRIPTION OF EMBODIMENTS

Hereinafter, example embodiments of this disclosure will be describedwith reference to the drawings. The same constituent elements are giventhe same reference numerals throughout all the drawings, and descriptionthereof will not be repeated as appropriate.

In each drawing of the present specification, a configuration of aportion having no relation to the essence of this disclosure is omittedand is not illustrated.

First Example Embodiment

FIG. 1 is a functional block diagram illustrating a logicalconfiguration of an information processing apparatus 100 according to anexample embodiment of this disclosure. The information processingapparatus 100 tracks a person who is not authenticated on the basis of acaptured image in a predetermined area.

The information processing apparatus 100 includes a collation unit 102and a registration unit 104. In the example illustrated in the presentfigure, the information processing apparatus 100 further includes astorage unit 110. The storage unit 110 includes a first storage unit 111(for example, a face feature information database) and a second storageunit 112. The storage unit 110 may be constituted by a plurality ofphysical storage media or a plurality of databases. The first storageunit 111 and the second storage unit 112 may be included in a physicallyidentical apparatus, and may be included in physically differentapparatuses. The first storage unit 111 and the second storage unit 112may be provided integrally with a main body of the informationprocessing apparatus 100, and may be provided separately therefrom.

The collation unit 102 collates first feature information extracted froma person included in a first image 20 with first registered featureinformation stored in the storage unit 110.

In a case where the first feature information cannot be extracted from aperson, or a collation result in the collation unit 102 indicates amismatch, the registration unit 104 stores second feature informationextracted from the person in the storage unit 110 as second registeredfeature information. In other words, the registration unit 104 registersthe second feature information of the person in the storage unit 110.

The collation unit 102 collates second feature information extractedfrom a person included in a second image 22 with the second registeredfeature information stored in the storage unit 110, and thus specifies aperson corresponding to the second registered feature information in thesecond image 22. Hereinafter, a collation process using the firstregistered feature information will be referred to as an “individualidentification process” in some cases, and a collation process using thesecond registered feature information will be referred to as a “personspecification process” in some cases.

The first feature information is feature information enabling a personto be specified, and is face feature information in the present exampleembodiment, and will be hereinafter referred to as face featureinformation in some cases. The first registered feature informationstored in the first storage unit 111 will also be hereinafter referredto as face registered feature information in some cases. The firstfeature information may be other biological information, for example, aniris, a pinna, a vein, a fingerprint, and a gait. The first storage unit111 may store a plurality of types of biological information for asingle person. In this case, the collation unit 102 may perform acollation process in combination of the plurality of types of biologicalinformation.

The second feature information is feature information of a personincluding regions other than the face, and will be hereinafter referredto as “person region feature information” in some cases. The secondregistered feature information registered in the second storage unit 112will be hereinafter referred to as registered person region featureinformation in some cases. The person region feature information isfeature information indicating an appearance feature such as a size orclothes of a person of which the face cannot be authenticated by thecollation unit 102. The person region feature information includesinformation indicating features such as a height, a shoulder width, abody part ratio, a garment (a shape, a color, a material, or the like),a hair style (also including a hair color), an ornament (a cap,spectacles, an accessory, or the like), and a belonging (a bag, anumbrella, or a stick). The person region feature information may includeinformation such as likelihood of the feature information.

Hereinafter, the information processing apparatus 100 will be describedin detail.

First, a description will be made of an image processed by theinformation processing apparatus 100. The first image 20 and the secondimage 22 may be captured by an identical imaging unit, and may becaptured by different imaging units. The imaging unit is a cameraincluding a lens and an imaging element such as a charge coupled device(CCD) image sensor, and is preferably a network camera such as anInternet Protocol (IP) camera. The network camera has, for example, awireless local area network (LAN) communication function, and isconnected to the information processing apparatus 100 through acommunication network, that is, a relay apparatus (not illustrated) suchas a router. The camera may be a so-called surveillance camera. Theimaging unit may include a mechanism which tracks movement of a specificperson in accordance with the movement, and performs control of a cameramain body or a direction of a lens, zoom control, or focusing.

The camera and the information processing apparatus 100 may be directlyconnected to each other, and may be indirectly connected to each otherthrough a communication network or the like as described above. Forexample, image data captured by the camera may be directly transmittedto the information processing apparatus 100, and the informationprocessing apparatus 100 may sequentially receive the image data. Astorage unit (not illustrated) which can be accessed by both of thecamera and the information processing apparatus 100 may be provided. Inthis case, image data captured by the camera may be stored in thestorage unit, and the information processing apparatus 100 may read andacquire the image data from the storage unit.

Here, the image data may be at least one of a still image and a movingimage. A data format, a file format, a file size, an image size, aresolution of an image, a frame rate of moving images, and the like arenot particularly limited, and data of various formats may be employedaccording to specifications, standards, performance, and the like of thecamera and the information processing apparatus 100, or image analysisprocessing performance or accuracy thereof. At least one frame of theimage data is at least one of the first image 20 and the second image22.

In the example embodiment, the “acquisition” includes at least one of anapparatus fetching (active acquisition) data or information stored inanother apparatus or a storage medium and the apparatus receiving(passive acquisition) data or information which is output from anotherapparatus. As an example of the active acquisition, there are a casewhere an apparatus sends a request or an inquiry to another apparatus,and receives a response thereto, and a case where the apparatus accessesanother apparatus or a storage medium, and reads data or information. Asan example of the passive acquisition, there is a case where anapparatus receives delivered information (alternatively, transmittedinformation or information sent through push notification). The“acquisition” may include selectively acquiring data or information fromreceived data or information, or selectively receiving delivered data orinformation.

Regarding a timing at which an image is transmitted from the camera tothe information processing apparatus 100, an image may be delivered inreal time, for example, through streaming delivery, and imagescorresponding to a predetermined period may be transmitted at apredetermined interval. The timing may be selected as appropriate on thebasis of a memory capacity, a communication capacity, or imageprocessing performance of the camera or the information processingapparatus 100, or a communication situation or the like between thecamera and the information processing apparatus 100, and may be changeddepending on a situation change.

In the present example embodiment, the first image 20 is captured at afirst timing, and the second image 22 is captured after the firsttiming.

The phrase “after the first timing” indicates either one of a timingafter a predetermined time elapses from the first timing and apredetermined time point later than the first timing. The second image22 captured “after the first timing” may be a plurality of imagescaptured at different timings. In this case, an individualidentification process and a person specification process in thecollation unit 102 are sequentially performed on each image. In otherwords, processes of performing an individual identification process onthe second image 22 having undergone a person specification process asthe first image 20 at the next timing, and of performing a personspecification process on the second image 22 captured at the furthernext timing, may be repeatedly performed.

As described above, the first storage unit 111 of the informationprocessing apparatus 100 stores the face feature information of anindividual who is an authentication target. For example, it is assumedthat face feature information of an employee is stored in the firststorage unit 111. In this case, in a case where face feature informationmatching face feature information detected from a captured image is notstored in the first storage unit 111, a person with the face may beestimated to be a person who is not an employee. It should be noted thata matching method in a collation process may employ various methods, andis not limited to a specific method.

For example, an individual identification process in the collation unit102 may be performed according to any of the following procedures, butis not limited thereto.

(a1) A person region is first detected, and a facial region in thedetected person region is specified. Authentication is performed on thespecified facial region.

(a2) A facial region is first detected, and authentication is performedon the detected facial region. A person region including the facialregion is specified. In this case, in a case where authentication ispossible, specification of a person region may be omitted.

(a3) A person region and a facial region are detected, andauthentication is performed on the detected facial region.

In addition to a case where there is no matching face featureinformation in an individual identification process, the registrationunit 104 may store person region feature information based on a personregion on which a collation process cannot be performed, in the secondstorage unit 112 in a case where a facial region cannot be determinedfrom a process included in the first image 20. For example, even in acase where a suspicious person recognizes a position of a surveillancecamera, and face feature information cannot be acquired as a result ofthe person having acted by avoiding the camera, person region featureinformation is stored in the second storage unit 112.

A collation result in the collation unit 102 includes either one ofinformation indicating that the first feature information (face featureinformation) or the second feature information (person region featureinformation) matches stored feature information and informationindicating that the first feature information or the second featureinformation does not match (mismatches) the stored feature information.The information indicating a mismatch with face feature information mayinclude a case where face feature information cannot be extracted from aperson included in the first image 20.

FIG. 2 is a diagram illustrating an example of a data structure of thesecond storage unit 112.

In the second storage unit 112, date-and-time information and personregion feature information are stored in association with each other.The date-and-time information is at least one of a capturing time pointof the first image 20, an acquisition time point of the first image 20,an execution time point of a face authentication process based on thefirst image 20, an acquisition time point of a result of the faceauthentication process, and a preservation time point of person regionfeature information.

The date-and-time information is stored in association with the personregion feature information in the second storage unit 112 in FIG. 2, butthe date-and-time information is not necessary, and may not be included.In an example of the present example embodiment, the second storage unit112 stores person region feature information of a person of which a facecannot be authenticated, and does not store person region featureinformation of a person of which a face has been authenticated, in anindividual identification process performed by the collation unit 102.Therefore, the present example embodiment is based on the fact thatperson members included in the first image 20 are not changed in thesecond image 22. It should be noted that a case where there is a changefrom person members detected by a camera 10 will be described in otherexample embodiments.

Hereinafter, a description will be made of an operation of theinformation processing apparatus 100 configured as mentioned above. FIG.3 is a flowchart illustrating an example of an operation of theinformation processing apparatus 100 of the present example embodiment.

First, a first image obtained by the camera 10 imaging a predeterminedarea is transmitted to the information processing apparatus 100 (stepS101). For example, it is assumed that three persons such as a person A,a person B, and a person C are captured in the first image.

In the information processing apparatus 100, the collation unit 102extracts face feature information from the persons included in the firstimage 20 received from the camera 10 (step S111). Here, three persons(the person A, the person B, and the person C) are detected, and facefeature information of each thereof is extracted. The collation unit 102performs a collation process (individual identification process) withface registered feature information stored in the first storage unit 111on each piece of the extracted face feature information (step S113).

Herein, it is assumed that, with respect to the person A, matching faceregistered feature information is included in the first storage unit 111(YES in step S115). Therefore, individual identification of the person Ais completed, processes in step S117 and the subsequent steps are notperformed, and the present process is finished. On the other hand,since, with respect to the person B and the person C, matching facefeature information is not included in the first storage unit 111 (NO instep S115), the flow proceeds to step S117, and person region featureinformation and date-and-time information corresponding to personregions of the person B and the person C are stored in the secondstorage unit 112. In other words, the person B and the person C aresubsequent tracking target persons.

The captured second image 22 is transmitted from the camera 10 to theinformation processing apparatus 100 after the first timing at which thefirst image 20 is acquired (step S103). In the information processingapparatus 100, the collation unit 102 receives the second image 22 fromthe camera 10, and detects a person region from the received secondimage 22 so as to extract person region feature information (step S119).

The collation unit 102 performs a process (person specification process)of collating the person region feature information extracted from thesecond image 22 with registered person region feature information storedin the second storage unit 112 (step S121).

In a case where registered person region feature information matchingthe extracted person region feature information is included in thesecond storage unit 112 (YES in step S123), the person B and the personC corresponding to the registered person region feature information arespecified in the second image 22 (step S125). The present process isfinished.

In the above-described way, the person B and the person C can be trackedin the second image 22.

On the other hand, even though a person (herein, the person A) of whichregistered person region feature information matching the extractedperson region feature information is not included in the second storageunit 112 (NO in step S123) is captured in the second image 22, theperson is not a tracking target, step S125 is bypassed, and the presentprocess is finished.

The process in the flowchart is repeatedly performed when an image istransmitted from the camera 10. With respect to the person B and theperson C of which person region feature information is stored in thesecond storage unit 112 in step S117, the processes in step S119 and thesubsequent steps may be repeatedly performed whenever the second image22 is received. In other words, in each second image 22, the person Band the person C corresponding to the person region feature informationstored in the second storage unit 112 are continuously tracked.

As described above, in the information processing apparatus 100 of thepresent example embodiment, the registration unit 104 stores personregion feature information of a person who cannot be identified in thefirst image 20 by the collation unit 102, in the second storage unit 112as information of an undetermined person. The collation unit 102collates person region feature information extracted from the secondimage 22 with registered person region feature information stored in thesecond storage unit 112, and thus a person corresponding to matchinginformation is specified. According to the configuration, with respectto a person undetermined in the first image 20, a person can bespecified on the basis of person region feature information in thesecond image 22 captured at a timing later than the first image 20, andthus the undetermined person can be continuously tracked.

As mentioned above, according to the present example embodiment, aperson who is not authenticated (that is, a person to which attention isrequired to be paid) can be tracked in a predetermined area after thefirst timing, and thus person region feature information of the personcan be stored as information of an undetermined person in the secondstorage unit 112. For example, in a case where a suspicious person orthe like who cannot be authenticated is mixed into a predetermined area,the person can be continuously tracked.

Second Example Embodiment

The information processing apparatus 100 of the present exampleembodiment is the same as the information processing apparatus 100 ofthe above-described example embodiment illustrated in FIG. 1 except thata person included in the second image 22 is specified, and then acollation process using face feature information extracted from theperson is performed. The information processing apparatus 100 of thepresent example embodiment has the same configuration as that of theinformation processing apparatus 100 of the above-described exampleembodiment illustrated in FIG. 1, and will thus be described withreference to FIG. 1.

The collation unit 102 collates face feature information extracted froma person specified in the second image 22 with face registered featureinformation stored in the first storage unit 111 of the storage unit110.

In other words, a face authentication process is performed on a personwho is continuously tracked in the second image 22, and thus individualidentification can be performed on an undetermined person. Even thoughthe face is turned away from the camera in the first image 20 capturedat the first timing, and thus face authentication is not performed well,in a case where the face is captured in the second image 22 capturedafter the first timing, there is a probability that facere-authentication may be successful.

FIG. 4 is a diagram illustrating an example of a data structure of thesecond storage unit 112 of the present example embodiment.

The second storage unit 112 further includes a determination flag inaddition to date-and-time information and person region featureinformation in the second storage unit 112 in FIG. 2. The determinationflag is a flag indicating that the person can be identified or cannot beidentified through a face authentication process in the collation unit102, may be set to “1” in a determined state, and may be set to “0” inan undetermined state. Alternatively, the determination flag may be setto “0” in a determined state, and may be set to “1” in an undeterminedstate. For example, a flag may be set only in a determined state, andNULL may be set in an undetermined state. Alternatively, a flag may beset only before determination, that is, in an undetermined state, andNULL may be set after determination. Alternatively, the execution dateand time of a face authentication process in which the person isidentified may be stored in the second storage unit 112 in associationwith person region feature information.

In the present example embodiment, in a case where matching occurs in acollation process with face feature information extracted from a personspecified in the second image 22, the registration unit 104 sets adetermination flag associated with registered person region featureinformation corresponding to the person stored in the second storageunit 112 to 1 from 0.

Alternatively, the information processing apparatus 100 may include adeletion unit (not illustrated) which deletes registered person regionfeature information of a person specified in the second image 22 fromthe second storage unit 112 in a case where face feature informationextracted from the person matches face registered feature informationstored in the first storage unit 111 as a collation result. In theconfiguration of having the deletion unit, a data structure of thesecond storage unit 112 is the same as the data structure illustrated inFIG. 2 of the above-described example embodiment.

FIG. 5 is a flowchart illustrating an example of an operation of theinformation processing apparatus 100 of the present example embodiment.

A flowchart of FIG. 5A includes step S131 to step S133 after step S125in the flowchart of FIG. 3.

The collation unit 102 extracts face feature information from the personspecified in the second image 22 in step S125 (step S131), performs acollation process (individual identification process) between theextracted face feature information and face registered featureinformation stored in the first storage unit 111 of the storage unit 110(step S133), and finishes the present process.

As illustrated in FIG. 5B, the collation unit 102 repeatedly performsprocesses (step S119 to step S125 in FIG. 3) of specifying a personcorresponding to the second registered feature information (registeredperson region feature information) in the second image 22 and processes(step S131 to step S133 in FIG. 5A) of collating the first featureinformation (face feature information) extracted from the specifiedperson with the first registered feature information (face registeredfeature information) until a collation result indicates a match (YES instep S135 in FIG. 5B) (the flow returns to step S119 in FIG. 3).

For example, of the person B and the person C who are being tracked asundetermined persons in the above-described example embodiment, withrespect to the person C, it is assumed that a face thereof is notcaptured in the first image 20, and thus face authentication fails inthe individual identification process in step S113 in FIG. 3. However,in a case where the face is captured in the second image 22 capturedlater than the first image 20, face authentication is successful in theindividual identification process in step S133 in FIG. 5A, and thus theperson C can be identified.

In the present example embodiment, the collation unit 102 may repeatedlyperform a collation process on a plurality of repeatedly captured secondimages 22 within a predetermined time or during a period till apredetermined time point.

A description will be made of two examples for a process on personregion feature information in which face authentication is successful inthe collation unit 102 and the person is identified such as an example(FIG. 6A) in which the second storage unit 112 in FIG. 4 is used, and anexample (FIG. 6B) in which the information processing apparatus 100 hasa deletion unit, and the second storage unit 112 in FIG. 2 is used.

First, a description will be made of the example illustrated in FIG. 6A.

In a case where face feature information extracted from a personspecified in step S133 matches any one of pieces of face registeredfeature information stored in the first storage unit 111 (YES in stepS135), the registration unit 204 sets 1 to the determination flagassociated with person region feature information corresponding to theperson of which the face has been authenticated in the second storageunit 112 in FIG. 4 (step S137). The present process is finished.

Herein, it is assumed that the person C can be identified (YES in stepS135), and 1 is set in the determination flag associated with personregion feature information corresponding to the person C (step S137).

On the other hand, in a case where the face feature informationextracted from the person specified in step S133 does not match any ofthe pieces of face registered feature information stored in the firststorage unit 111 (NO in step S135), step S137 is bypassed, and thepresent process is finished. In other words, the determination flagassociated with person region feature information in the second storageunit 112 in FIG. 4 is still set to 0. In this example, it is assumedthat the person B fails in face authentication (NO in step S135), stepS137 is bypassed, and the present process is finished. Therefore, thedetermination flag associated with the person region feature informationof the person B is still set to 0.

In the example illustrated in FIG. 6B, in a case where face featureinformation extracted from a person specified in step S133 matches anyone of pieces of face registered feature information stored in the firststorage unit 111 (YES in step S135), the deletion unit deletes personregion feature information corresponding to the person of which the facehas been authenticated from the second storage unit 112 in FIG. 2.

Herein, it is assumed that matching face feature information of theperson C is included in the first storage unit 111 (YES in step S135),and the deletion unit deletes the person region feature informationcorresponding to the person C from the second storage unit 112 in FIG. 2(step S137).

On the other hand, in a case where the face feature informationextracted from the person specified in step S133 does not match any ofthe pieces of face registered feature information stored in the firststorage unit 111 (NO in step S135), step S139 is bypassed, and thepresent process is finished. In other words, the person region featureinformation is still stored in the second storage unit 112.

Herein, it is assumed that matching face feature information of theperson B is not included in the first storage unit 111 (NO in stepS135), step S139 is bypassed, and the present process is finished. Asmentioned above, in a case of the second storage unit 112 in FIG. 2,only the person region feature information of the person B is stillstored in the second storage unit 112.

As described above, according to the present example embodiment, sincethe collation unit 102 performs a collation process between face featureinformation extracted from a person specified in the second image 22 andface registered feature information stored in the first storage unit111, even a person of which a face cannot be authenticated in the firstimage 20 succeeds in face authentication in the second image 22 capturedat a timing later than the first image 20, and thus the person can beidentified.

Since information (determination flag) indicating whether or not a facehas been authenticated is stored in the second storage unit 112 inassociation with person region feature information, or person regionfeature information of a person of which a face has been authenticatedis deleted from the second storage unit 112, it is possible to acquireperson region feature information of a person of which a face cannot beauthenticated within a predetermined time or by a predetermined timepoint, by referring to the second storage unit 112.

Third Example Embodiment

FIG. 7 is a functional block diagram illustrating a logicalconfiguration of an information processing apparatus 100 of the presentexample embodiment.

The information processing apparatus 100 of the present exampleembodiment is the same as the information processing apparatus 100 ofthe above-described example embodiment in FIG. 1 except that an outputunit 120 is further provided in addition to the configuration thereof.

In the present example embodiment, the registration unit 104 storesperson region feature information in the second storage unit 112 alongwith time information. The output unit 120 outputs information regardingregistered person region feature information stored in the secondstorage unit 112 according to a predetermined condition.

In an example of a first predetermined condition, in a case where facefeature information cannot be extracted from a person specified throughcollation with registered person region feature information in thesecond image 22 even though a reference time or more elapses from a timepoint indicated by the time information, or in a case where a collationresult with face feature information extracted from a person specifiedthrough collation with registered person region feature information inthe second image 22 indicates a mismatch, the output unit 120 outputsinformation regarding the registered person region feature information.

Here, the time information corresponds to the date-and-time informationin the second storage unit 112 in FIG. 2 or 4, and is at least one of atime point at which person region feature information is stored in thesecond storage unit 112, a time point at which a collation target imageis captured (or stored), a time point at which a collation target imageis received, a time point at which a collation process is performed, anda time point at which a collation result is stored in the second storageunit 112.

In an example of a second predetermined condition, the output unit 120outputs information regarding registered person region featureinformation which is stored in the second storage unit 112 at areference timing. In a case of the second storage unit 112 in FIG. 2described in the above-described example embodiment, a personcorresponding to registered person region feature information remainingin the second storage unit 112 at a reference timing is an undeterminedperson who is not identified by the timing. Therefore, the output unit120 displays information regarding the registered person region featureinformation, for example, an image of a person region corresponding tothe registered person region feature information, and can thus make anoperator recognize an undetermined person.

The reference timing is a predetermined time point, the last timing, oran imaging timing in a camera at a predetermined position, or the like.

The output unit 120 displays a captured image on, for example, adisplay, and also displays a mark indicating whether or notauthentication is completed to overlap each person. For example, a markof “OK” is displayed for an authenticated person, and a mark of “NG” isdisplayed for an unauthenticated person. The “OK” mark may be displayedfor only an authenticated person, or the “NG” mark may be displayed foronly an unauthenticated person. An image (for example, person regionsare surrounded by rectangular frames with different colors) in whichperson regions of an authenticated person and an unauthenticated personcan be distinguished from each other is drawn to overlap an image inwhich registered person region feature information is extracted. Aperson region of an unauthenticated person may be displayed in anemphasis manner.

In addition to screen display, an output form of the output unit 120 maybe to output an alert sound from a speaker, to transmit a message to apredetermined destination or terminal, to cause a light emitting diode(LED) indicator to blink or to light, or to output information to aprinter such that the information is printed. The output unit 120 mayoutput a locking or unlocking control signal to a locking apparatus ofan entrance door in a predetermined area, or may output an opening orclosing control signal to a drive apparatus of a gate, an automaticdoor, or a shutter. In other words, in a case where person regionfeature information of a person who is not authenticated remains in thesecond storage unit 112 after a reference time elapses or at a referencetiming, a control signal for locking a door or closing a gate may beoutput to each apparatus such that the person corresponding to theperson region feature information cannot enter or exit the area.

The output unit 120 may output, to a camera, a control signal forincreasing image quality or a resolution of a captured image including aperson region of a person who is not authenticated, or a control signalfor tracking or zooming in the person region.

The information regarding the registered person region featureinformation may include an image of a person region corresponding to theregistered person region feature information. The registered personregion feature information may include information indicating whether ornot authentication is completed, and information regarding a time pointat which an image is captured or a time point at which faceauthentication is performed. In a case where there are a plurality ofsurveillance target areas, identification information of an area may beincluded. In a case where a plurality of cameras 10 are used,identification information of the camera 10 having captured an imageincluding a person region corresponding to the registered person regionfeature information may be included.

FIG. 8 is a diagram for explaining a timing of information output in theoutput unit 120. FIG. 8C will be described in an example embodimentwhich will be described later.

In an example illustrated in FIG. 8A, the output unit 120 outputs analert for person region feature information which has been stored in thesecond storage unit 112 for a reference time or more from a time pointof a timing at which the person region feature information is stored forthe first time. In this example, it is possible to monitor whether ornot face authentication can be performed within a reference time foreach person.

In an example illustrated in FIG. 8B, the output unit 120 outputs analert for person region feature information remaining at a certainreference timing, for example, a predetermined time point with respectto person region feature information stored in the second storage unit112.

As described above, in the present example embodiment, the registrationunit 104 stores person region feature information in the second storageunit 112 along with time information, and the output unit 120 outputsinformation regarding registered person region feature information of aperson of which a face is not authenticated even though a reference timeor more elapses from a time point indicated by the time information.According to the present example embodiment, it is possible to achievethe same effect as in the above-described example embodiment, and alsoto perform a notification of a person who cannot be authenticated in acertain area for a predetermined time or more or by a predetermined timepoint.

In the present example embodiment, the output unit 120 outputsinformation regarding registered person region feature informationstored in the second storage unit 112 at a reference timing. Accordingto the present example embodiment, it is possible to perform for eachpredetermined time point a notification of a person which cannot beauthenticated.

Fourth Example Embodiment

The information processing apparatus 100 of the present exampleembodiment is the same as the information processing apparatus 100 ofthe above-described example embodiment except that pieces of personregion feature information of all persons extracted from the first imageare stored in the second storage unit 112, a face authentication processis repeatedly performed until the face authentication process issuccessful while tracking a person by using registered person regionfeature information. The information processing apparatus 100 of thepresent example embodiment has the same configuration as that of theinformation processing apparatus 100 of the above-described exampleembodiment illustrated in FIG. 1, and will thus be described withreference to FIG. 1. The information processing apparatus 100 of thepresent example embodiment may have the same configuration as that ofthe information processing apparatus 100 of the other exampleembodiments, and the configurations may be combined with each otherwithin the scope without contradiction.

The present example embodiment is different from the above-describedexample embodiments in terms of the following contents.

The registration unit 104 stores person region feature informationextracted from a person included in the first image 20 in the secondstorage unit 112 as registered person region feature information.

The collation unit 102 collates person region feature informationextracted from a person included in the second image 22 with registeredperson region feature information stored in the second storage unit 112,and thus specifies a person corresponding to the registered personregion feature information in the second image 22. In a case where facefeature information cannot be extracted from a person included in thefirst image 20, or a collation result between face feature informationextracted from a person included in the first image 20 with the firstregistered feature information (face registered feature information)indicates a mismatch, the collation unit 102 collates face featureinformation extracted from a person specified in the second image 22with face registered feature information stored in the first storageunit 111.

The registration unit 104 may store information indicating a collationresult between face feature information extracted from a person includedin the first image 20 and face registered feature information in thesecond storage unit 112 in association with registered person regionfeature information. Alternatively, the registration unit 104 may storeinformation indicating that face feature information cannot be extractedfrom a person included in the first image 20 in the second storage unit112 in association with registered person region feature information.

FIG. 9 is a diagram illustrating an example of a data structure of thesecond storage unit 112 of the present example embodiment.

The second storage unit 112 stores date-and-time information, personregion feature information, a collation result flag in association witheach other. The date-and-time information is at least one of a capturingtime point of the first image 20, an acquisition time point of the firstimage 20, an execution time point of an authentication process based onthe first image 20, an acquisition time point of a result of theauthentication process, and a preservation time point of registeredperson region feature information.

The collation result flag is information indicating a collation resultof face feature information in the collation unit 102. The informationindicating a collation result includes at least one of informationindicating a match with face registered feature information andinformation indicating a mismatch with face registered featureinformation.

In a case where there is matching face registered feature information inthe first storage unit 111 in a collation process, that is, the personcan be identified, for example, “1” is set in the collation result flag.In a case where there is no matching face registered featureinformation, for example, “0” is set in the collation result flag. Forexample, only in a case of a match, the collation result flag may beset, and, in a case where there is no matching face feature information,NULL may be set. Alternatively, the collation result flag may be set to“0” in a case of a match, and may be set to “1” in a case of a mismatch.The collation result flag may be set in a case of a mismatch, and NULLmay be set in a case of a match.

In a case where the information indicating a collation result indicatesthat a collation result with face feature information extracted from aperson included in the first image 20 indicates a mismatch, thecollation unit 102 collates face feature information extracted from aperson specified in the second image 22 with face registered featureinformation stored in the first storage unit 111.

In a case where information associated with registered person regionfeature information indicates that face feature information cannot beextracted from a person included in the first image 20, the collationunit 102 collates face feature information extracted from a personspecified in the second image 22 with face registered featureinformation stored in the first storage unit 111.

The information processing apparatus 100 of the present exampleembodiment may include the same output unit 120 as that of theinformation processing apparatus 100 of the above-described exampleembodiment illustrated in FIG. 7.

FIG. 10 is a flowchart illustrating an example of an operation of theinformation processing apparatus 100 of the present example embodiment.

It should be noted that, in FIG. 10, the same process procedure as inthe above-described example embodiment is given the same step number.

First, the first image 20 obtained by the camera 10 imaging apredetermined area is transmitted to the information processingapparatus 100 (step S101). For example, it is assumed that three personssuch as a person A, a person B, and a person C are captured in the firstimage 20.

In the information processing apparatus 100, the collation unit 102extracts face feature information from the persons included in the firstimage 20 received from the camera 10 (step S111). Here, three persons(the person A, the person B, and the person C) are detected, and facefeature information of each thereof is extracted. The collation unit 102performs a collation process (individual identification process) withface registered feature information stored in the first storage unit 111on each piece of the extracted face feature information (step S113).

Herein, it is assumed that matching face registered feature informationof the person A is included in the first storage unit 111. In otherwords, since individual identification of the person A is completed, theregistration unit 104 sets “1” in a collation result flag for the personA as information indicating a match, and stores the collation resultflag in the second storage unit 112 in association with person regionfeature information and date-and-time information (step S201).

On the other hand, it is assumed that, with respect to the person B andthe person C, matching face feature information is not included in thefirst storage unit 111. Therefore, the registration unit 104 sets “0” incollation result flags for the person B and the person C as informationindicating a mismatch, and stores the collation result flags in thesecond storage unit 112 in association with person region featureinformation and date-and-time information (step S201).

The captured second image 22 is transmitted from the camera 10 to theinformation processing apparatus 100 after the first timing at which thefirst image 20 is acquired (step S103). In the information processingapparatus 100, the collation unit 102 receives the second image 22 fromthe camera 10, and detects a person region from the received secondimage 22 so as to extract person region feature information (step S119).

The collation unit 102 performs a process (person specification process)of collating the person region feature information extracted from thesecond image 22 with registered person region feature information storedin the second storage unit 112 (step S121). In a case where registeredperson region feature information matching the extracted person regionfeature information is included in the second storage unit 112 (YES instep S123), a person corresponding to the registered person regionfeature information is specified in the second image 22 (step S125). Inthis example, the three persons stored in the second storage unit 112,that is, the person A, the person B, and the person C are specified.

In a case where a collation result flag indicating that a collationresult of face feature information of each person indicates a match isnot set to “1” (NO in step S203) by referring to the second storage unit112, the collation unit 102 proceeds to step S205. In a case where acollation result indicates a match, that is, the collation result flagis set to “1” (YES in step S203), step S205 to step S209 are bypassed,and the present process is finished. Herein, a collation result flag forthe person A is 1, and thus the process for the person A is finished.Since collation result flags for the person B and the person C are 0,the flow proceeds to step S205.

The collation unit 102 extracts face feature information from eachspecified person (step S205). Herein, face feature information of eachof two persons (the person B and the person C) is extracted. Thecollation unit 102 performs a collation process (individualidentification process) with the face registered feature informationstored in the first storage unit 111 on each piece of the extracted facefeature information (step S207).

Herein, it is assumed that matching face feature information of theperson C is included in the first storage unit 111. In other words,since individual identification of the person C is completed, theregistration unit 104 sets “1” in a collation result flag for the personC as information indicating a match, stores the collation result flag inassociation with person region feature information and date-and-timeinformation corresponding to the person C (step S209), and finishes thepresent process.

On the other hand, it is assumed that matching face feature informationof the person B is not included in the first storage unit 111. Theregistration unit 104 sets “0” in a collation result flag for the personB as information indicating a mismatch, stores the collation result flagin association with person region feature information and date-and-timeinformation corresponding to the person B (step S209), and finishes thepresent process.

In the above-described way, it is possible to perform facere-authentication of a person of which a face cannot be authenticatedwhile continuously tracking each person (the person A, the person B, andthe person C) by using the second image 22. It is possible to identifythe person by performing face authentication by using an image capturedwhen a person with a face not initially directed to a camera directs theface to the camera thereafter.

In step S209, both of the person region feature information stored inthe second storage unit 112 and the person region feature informationalready stored in the second storage unit 112 in step S201 may be storedin association with each other. Alternatively, the person region featureinformation specified in step S207 may be overwritten on the personregion feature information in the second storage unit 112, and only thecollation result flag may be updated by leaving the person regionfeature information already stored in the second storage unit 112 instep S201.

In the process in the flowchart, the processes in step S119 and thesubsequent steps are repeatedly performed when the second image 22 istransmitted from the camera 10.

In a case where a new person D not included in the first image 20 isdetected in the second image 22, that is, person region featureinformation matching person region feature information extracted fromthe second image 22 is not stored in the second storage unit 112 (NO instep S123), the flow may return to step S111 with the second image 22 asthe first image 20, and the process may be repeatedly performed on theperson region feature information, corresponding to the new person D.

In the present example embodiment, step S131 to step S133 in FIG. 5A,step S135 in FIG. 5B, and step S135 to step S137 in FIG. 6A, or stepS135 to step S139 in FIG. 6B may be executed after step S209.

As described above, in the present example embodiment, the registrationunit 104 stores person region feature information extracted from aperson included in the first image 20 in the second storage unit 112 inassociation with information indicating a collation result of faceauthentication in the collation unit 102. Each person corresponding toperson region feature information stored in the second storage unit 112can be tracked by being specified in the second image 22 captured at atiming later than the first image 20. Among persons specified in thesecond image 22, the collation unit 102 performs a face authenticationprocess on a person who is not identified yet through faceauthentication.

According to the present example embodiment, it is possible to achievethe same effect as that in the above-described example embodiments, andalso to identify a person of which face authentication is not completedby performing face authentication on the person while tracking a personincluded in an image by using person region feature information.

In the present example embodiment, the registration unit 104 storesperson region feature information in the second storage unit 112 inassociation with collation results of face authentication for allpersons included in each image. According to this configuration, even ina case where persons enter and exit a predetermined area, it is possibleto continuously track each person and to perform an individualidentification process. It is also possible to count and check thenumber of persons present in a predetermined area, and to check a personhaving entered the area or having exited the area along with timeinformation.

Fifth Example Embodiment

The information processing apparatus 100 of the present exampleembodiment is the same as that of any one of the first to third exampleembodiments except for a configuration for finding a person who is notauthenticated between a start point of a predetermined path to an endpoint thereof by using captured images from a plurality of camerasprovided along the path unlike the above-described example embodiments.

The information processing apparatus 100 of the present exampleembodiment has the same configuration as that of the informationprocessing apparatus 100 in FIG. 1, and will thus be described withreference to FIG. 1.

The information processing apparatus 100 processes an image generated byeach of a plurality of cameras 10 provided along a path.

The path includes at least two different points (for example, a startpoint and an end point), and is a route along which a person passesbetween the two points. The path may be, for example, a route alongwhich a person passes from entry through an entrance of a certainfacility, building, or area to exit through an exit thereof, an approachalong which a person enters a certain site through an entrance thereofand passes until entering a building through a doorway thereof, apassage along which a person moves from a certain area to another area,or a path along which a person passes from entry through a gate of acertain station to exit through a gate of another station. A route alongwhich a person passes between two points may differ in each person.

A plurality of cameras provided along a path include at least twocameras such as a first camera provided at a start point through which aperson enters the path and a second camera provided at an end pointthrough which the person exits the path.

In the present example embodiment, the first image 20 is an imagecaptured by the first camera. The second image 22 is an image capturedby the second camera located after the first camera in a direction alongthe path. The second image 22 is captured at a timing after a firsttiming at which the first image 20 is captured. It should be noted thata direction along the path may differ in each person.

Specifically, the collation unit 102 collates face feature informationextracted from a person included in the first image 20 with faceregistered feature information stored in the first storage unit 111 ofthe storage unit 110.

In a case where face feature information cannot be extracted from aperson, or a collation result in the collation unit 102 indicates amismatch, the registration unit 104 stores person region featureinformation extracted from the person in the second storage unit 112 ofthe storage unit 110 as registered person region feature information.

The collation unit 102 collates person region feature informationextracted from the person included in the second image 22 with theregistered person region feature information stored in the secondstorage unit 112, and thus specifies the person corresponding to theregistered person region feature information in the second image 22.

Among a plurality of cameras provided along the path, the second camerais provided in a plurality at a predetermined interval or atpredetermined positions between the start point to the end point alongthe path.

The collation unit 102 performs a collation process with the personregion feature information stored in the second storage unit 112 on thesecond images 22 captured by the plurality of second cameras, and thusspecifies the person corresponding to the person region featureinformation. Consequently, the information processing apparatus 100 ofthe present example embodiment can track each person region (person)passing along the path in a plurality of images captured time pointswhich are different from the first timing, captured at differentpositions, or captured by a plurality of different second cameras.

The second storage unit 112 of the present example embodiment may havethe data structure in FIG. 2 or 4.

The information processing apparatus 100 of the present exampleembodiment may include the same output unit 120 as that in the thirdexample embodiment illustrated in FIG. 7. The output unit 120 of thepresent example embodiment is different from the output unit 120 of thethird example embodiment in terms of the following output conditions. Itshould be noted that the output unit 120 of the present exampleembodiment may have a configuration using a combination with the samecondition as in the output unit 120 of the third example embodiment.

An output condition of the output unit 120 is that there is no matchwith face feature information in a collation process performed by thecollation unit 102 with respect to a person who is specified by thecollation unit 102 by using an image generated by the second cameraprovided last (end point) or at a predetermined position in a directionalong the path among a plurality of second cameras.

In a case where a collation result with face registered featureinformation indicates a mismatch with respect to a person regionspecified by the collation unit 102 in an image generated by the secondcamera provided last (end point) or at a predetermined position in adirection along the path among a plurality of second cameras, the outputunit 120 outputs information regarding person region feature informationextracted from the person.

The registration unit 104 may store person region feature information inthe second storage unit 112 along with time information, and the outputunit 120 may output information regarding person region featureinformation of which a collation result with face registered featureinformation stored in the second storage unit 112 indicates a mismatcheven though a reference time or more elapses from a time point indicatedby the time information.

Here, the case where a collation result indicates a mismatch includes acase where face feature information cannot be extracted from a personspecified in the second image 22 or a case where a collation resultbetween face feature information extracted from a person specified inthe second image 22 and face registered feature information indicates amismatch.

In addition to the output form of the output unit 120 of theabove-described example embodiment, the output unit 120 of the presentexample embodiment may output, for example, an opening or closingcontrol signal to a drive apparatus of gates provided at both ends or inthe middle of the path. In other words, in a case where there is nomatch with face feature information in a collation process performed bythe collation unit 102 with respect to a person who is specified by thecollation unit 102 by using an image generated by the second cameraprovided at a predetermined location on the path, a control signal forclosing a gate provided in the middle of the path or provided at adoorway may be output to a grate drive apparatus such that the personcannot pass along the path.

As illustrated in FIG. 8C, the output unit 120 outputs informationregarding person region feature information remaining in the secondstorage unit 112 at a time point at which a person having entered thepath through an entrance exits the path through an exit.

FIG. 11 is a flowchart illustrating an example of an operation of theinformation processing apparatus 100 of the present example embodiment.

First, the first image 20 obtained by a first camera (referred to as afirst camera 10 a in some cases) provided at an entrance of a pathimaging a predetermined area is transmitted to the informationprocessing apparatus 100 (step S101). For example, it is assumed thatthree persons such as a person A, a person B, and a person C arecaptured in the first image 20.

In the information processing apparatus 100, the collation unit 102extracts face feature information from the persons included in the firstimage 20 received from the first camera 10 a (step S111). Here, threepersons (the person A, the person B, and the person C) are detected, andface feature information of each thereof is extracted. The collationunit 102 performs a collation process (individual identificationprocess) with face registered feature information stored in the firststorage unit 111 on each piece of the extracted face feature information(step S113).

Herein, it is assumed that matching face registered feature informationof the person A is included in the first storage unit 111 (YES in stepS115). Therefore, individual identification of the person A iscompleted, processes in step S117 and the subsequent steps are notperformed, and the present process is finished. On the other hand,since, with respect to the person B and the person C, matching facefeature information is not included in the first storage unit 111 (NO instep S115), the flow proceeds to step S117, and person region featureinformation and date-and-time information corresponding to the person Band the person C are stored in the second storage unit 112.

The captured second image 22 is transmitted from a second camera(referred to as a second camera 10 b in some cases) which is locatedafter the first camera 10 a in a direction along the path, to theinformation processing apparatus 100 after the first timing at which thefirst image 20 is acquired (step S103). In the information processingapparatus 100, the collation unit 102 receives the second image 22 fromthe second camera 10 b, and detects a person region from the receivedsecond image 22 so as to extract person region feature information (stepS119).

The collation unit 102 performs a process (person specification process)of collating the person region feature information extracted from thesecond image 22 with registered person region feature information storedin the second storage unit 112 (step S121). In a case where registeredperson region feature information matching the extracted person regionfeature information is included in the second storage unit 112 (YES instep S123), the person B and the person C corresponding to theregistered person region feature information are specified in the secondimage 22 (step S125). The present process is finished.

In the above-described way, the person B and the person C can be trackedin the second image 22.

On the other hand, a person (herein, the person A) of which registeredperson region feature information matching the extracted person regionfeature information is not included in the second storage unit 112 (NOin step S123) is not a tracking target, step S125 is bypassed, and thepresent process is finished.

The process in the flowchart is repeatedly performed when an image istransmitted from each camera 10. In step S117, with respect to theperson B and the person C of which person region feature information isstored in the second storage unit 112, the processes in step S119 andthe subsequent steps may be repeatedly performed whenever the secondimage 22 is received. In other words, in each second image 22, theperson B and the person C corresponding to the person region featureinformation stored in the second storage unit 112 are continuouslytracked.

As described above, in the information processing apparatus 100 of thepresent example embodiment, the registration unit 104 stores, asinformation of an undetermined person, person region feature informationof a person region in which a face authentication process cannot beperformed on the first image 20 which is captured by the first camera 10a at the first timing by using captured images from a plurality ofcameras provided along the path, in the second storage unit 112. Thecollation unit 102 specifies a person matching person region featureinformation stored in the second storage unit 112 in the second image 22which is captured by the second camera 10 b located after of the firstcamera 10 a in a direction along the path after the first timing.

As mentioned above, according to the present example embodiment, it ispossible to store person region feature information of a person whocannot be identified through face authentication from entry into thepath to exit from the path, in the second storage unit 112 asinformation of an undetermined person. For example, in a case where aperson of which face feature information is not stored in the firststorage unit 111 and who is not originally permitted to pass along apath tries to pass along the path, person region feature information ofthe person can be stored in the second storage unit 112 and be tracked.

In the present example embodiment, the registration unit 104 storesperson region feature information in the second storage unit 112 alongtime information of a timing at which the person region featureinformation is stored for the first time, and the output unit 120outputs information regarding person region feature information whichhas been stored in the second storage unit 112 for a reference time ormore.

Alternatively, in the present example embodiment, the output unit 120outputs information regarding person region feature information of aperson in a case where a collation process with face feature informationindicates a mismatch with respect to the person who is specified by thecollation unit 102 by using an image generated by the second cameraprovided last or at a predetermined position in a direction along thepath among a plurality of second cameras.

According to this configuration, among persons passing along the path,it is possible to perform a notification of information regarding aperson who cannot be identified through face authentication at alocation where the person comes out of an exit of the path or at apredetermined position. For example, face registered feature informationof a person who is permitted to pass along a path is stored in the firststorage unit 111, and it is possible to perform a notification ofpassing of a person of which a collation result with face registeredfeature information stored in the first storage unit 111 indicates amismatch, that is, the person who is not permitted to pass along thepath, at an exit of the path along which the person is required to bepermitted to pass.

Sixth Example Embodiment

The information processing apparatus 100 of the present exampleembodiment stores, in the second storage unit 112, person region featureinformation extracted from a person included in the first image 20captured by a first camera among a plurality of cameras provided along apath regardless of a collation result of face authentication, andspecifies and tracks a person by using the person region featureinformation stored in the second storage unit 112 in the second image 22captured by a second camera located after the first camera in adirection along the path.

The information processing apparatus 100 of the present exampleembodiment has the same configuration as that of the informationprocessing apparatus 100 of the fourth example embodiment except that aprocess is performed by using the first image 20 captured by a firstcamera among a plurality of cameras provided along a path and the secondimage 22 captured by a second camera located after the first camera in adirection along the path.

The collation unit 102 collates face feature information extracted froma person included in the first image 20 captured by the first cameraamong a plurality of cameras provided along a path, with face registeredfeature information stored in the first storage unit 111. Theregistration unit 104 stores the person region feature informationextracted from the person in the second storage unit 112 (FIG. 9). Thecollation unit 102 collates person region feature information extractedfrom a person included in the second image 22 generated by each of aplurality of second cameras located after the first camera in adirection along the path, with registered person region featureinformation stored in the second storage unit 112, so as to specify aperson in the second image 22, and tracks the person.

In a case where face feature information cannot be extracted from aperson included in the first image 20, or a collation result betweenface feature information extracted from a person included in the firstimage 20 with face registered feature information indicates a mismatch,the collation unit 102 collates face feature information extracted froma person specified in the second image 22 with face registered featureinformation stored in the first storage unit 111.

A description will be made of an operation of the information processingapparatus 100 of the present example embodiment having theconfiguration. FIG. 12 is a flowchart illustrating an example of anoperation of the information processing apparatus 100 of the presentexample embodiment. In the following description, the second storageunit 112 will be described to have the data structure in FIG. 9, but acollation result flag is not necessarily required to be used, and thesecond storage unit 112 may include at least person region featureinformation.

First, the first image 20 obtained by the first camera 10 a provided atan entrance of a path imaging a predetermined area is transmitted to theinformation processing apparatus 100 (step S101). For example, it isassumed that three persons such as a person A, a person B, and a personC are captured in the first image 20.

In the information processing apparatus 100, the collation unit 102detects the three persons (the person A, the person B, and the person C)the first camera 10 a among the plurality of cameras 10, and extractsface feature information of each person. The collation unit 102 performsa collation process (individual identification process) with faceregistered feature information stored in the first storage unit 111 oneach piece of the extracted face feature information (step S113).

Herein, it is assumed that matching face registered feature informationof the person A is included in the first storage unit 111. In otherwords, since individual identification of the person A is completed, theregistration unit 104 sets “1” in a collation result flag for the personA as information indicating a match, and stores the collation resultflag in the second storage unit 112 in association with person regionfeature information and date-and-time information (step S201).

On the other hand, it is assumed that, with respect to the person B andthe person C, matching face feature information is not included in thefirst storage unit 111. Therefore, the registration unit 104 sets “0” incollation result flags for the person B and the person C as informationindicating a mismatch, and stores the collation result flags in thesecond storage unit 112 in association with person region featureinformation and date-and-time information (step S201).

The captured second image 22 is transmitted from the second camera 10 bwhich is located after the first camera 10 a in a direction along thepath, to the information processing apparatus 100 after the first timingat which the first image 20 is acquired (step S103). In the informationprocessing apparatus 100, the collation unit 102 receives the secondimage 22 from the second camera 10 b, and detects a person region fromthe received second image 22 so as to extract person region featureinformation (step S119).

The collation unit 102 performs a process (person specification process)of collating the person region feature information extracted from thesecond image 22 with registered person region feature information storedin the second storage unit 112 (step S121). In a case where registeredperson region feature information matching the extracted person regionfeature information is included in the second storage unit 112 (YES instep S123), the person B and the person C corresponding to theregistered person region feature information are specified in the secondimage 22 (step S125). In this example, three persons such as the personA, the person B, and the person C stored in the second storage unit 112are specified.

In a case where a collation result flag indicating that a collationresult of face feature information of each person indicates a match isnot set to “1” (NO in step S203) by referring to the second storage unit112, the collation unit 102 proceeds to step S205. In a case where acollation result indicates a match, that is, the collation result flagis set to “1” (YES in step S203), step S205 to step S209 are bypassed,and the present process is finished. Herein, a collation result flag forthe person A is 1, and thus the process for the person A is finished.Since collation result flags for the person B and the person C are 0,the flow proceeds to step S205.

The collation unit 102 extracts face feature information from eachspecified person (step S205). Herein, face feature information of eachof two persons (the person B and the person C) is extracted. Thecollation unit 102 performs a collation process (individualidentification process) with the face registered feature informationstored in the first storage unit 111 on each piece of the extracted facefeature information (step S207).

Herein, it is assumed that matching face feature information of theperson C is included in the first storage unit 111. In other words,since individual identification of the person C is completed, theregistration unit 104 sets “1” in a collation result flag for the personC as information indicating a match, stores the collation result flag inassociation with person region feature information and date-and-timeinformation corresponding to the person C (step S209), and finishes thepresent process.

On the other hand, it is assumed that matching face feature informationof the person B is not included in the first storage unit 111. Theregistration unit 104 sets “0” in a collation result flag for the personB as information indicating a mismatch, stores the collation result flagin association with person region feature information and date-and-timeinformation corresponding to the person B (step S209), and finishes thepresent process.

In the above-described way, it is possible to perform facere-authentication of a person of which a face cannot be authenticatedwhile continuously tracking each person (the person A, the person B, andthe person C) by using the second images 22 which are captured by usinga plurality of cameras on the path. Since a face authentication processis repeatedly performed by using a plurality of images captured by aplurality of cameras while persons are passing along a path, it ispossible to identify a person who initially does not direct the face toa camera by performing face authentication by using an image capturedwhen the person directs the face to the camera next.

In step S209, both of the person region feature information stored inthe second storage unit 112 and the person region feature informationalready stored in the second storage unit 112 in step S201 may be storedin association with each other. Alternatively, the person region featureinformation specified in step S207 may be overwritten on the personregion feature information in the second storage unit 112, and only thecollation result flag may be updated by leaving the person regionfeature information already stored in the second storage unit 112 instep S201.

In the process in the flowchart, the processes in step S119 and thesubsequent steps are repeatedly performed whenever the second image 22is transmitted from the camera 10.

In a case where a new person D not included in the first image 20 isdetected in the second image 22, that is, person region featureinformation matching person region feature information extracted fromthe second image 22 is not stored in the second storage unit 112 (NO instep S123), the flow may return to step S111 with the second image 22 asthe first image 20, and the process may be repeatedly performed on theperson region feature information corresponding to the new person D.

In the present example embodiment, step S131 to step S133 in FIG. 5A,step S135 in FIG. 5B, and step S135 to step S137 in FIG. 6A, or stepS135 to step S139 in FIG. 6B may be executed after step S209.

According to this configuration, even in a case where a person turnsback in the middle of a path, the person can be specified and be trackedby using the second image 22 captured after the first image 20 iscaptured and person region feature information stored in the secondstorage unit 112.

Also in this configuration, the output unit 120 described with referenceto FIG. 7 may be provided.

Specifically, the output unit 120 outputs information regarding personregion feature information of a person in a case where there is no matchwith face registered feature information stored in the first storageunit 111 with respect to the person who is specified by the collationunit 102 by using an image generated by a camera provided last or at apredetermined position on the path among a plurality of cameras.

FIG. 13 is a diagram illustrating an example of a configuration of acomputer 80 realizing the information processing apparatus of each ofthe above-described example embodiments.

The computer 80 includes a central processing unit (CPU) 82, a memory84, a program 90, loaded to the memory 84, for implementing theconstituent elements of each information processing apparatus in FIGS. 1and 7, a storage 85 storing the program 90, an input/output (I/O) 86,and a network connection interface (communication I/F 87).

The CPU 82, the memory 84, the storage 85, the I/O 86, and thecommunication I/F 87 are connected to each other through a bus 89, andthe entire information processing apparatus is controlled by the CPU 82.However, a method of connecting the CPU 82 and the like to each other isnot limited to bus connection.

The memory 84 is a memory such as a random access memory (RAM) or a readonly memory (ROM). The storage 85 is a storage unit such as a hard disk,a solid state drive (SSD), or a memory card.

The storage 85 may be a memory such as a RAM or a ROM. The storage 85may be provided in the computer 80, may be provided outside the computer80 as long as the computer 80 can assess the storage, and may beconnected to the computer 80 in a wired or wireless manner.Alternatively, the storage may be provided to be attachable to anddetachable from the computer 80.

The CPU 82 reads the program 90 stored in the storage 85 to the memory84 and executes the program, and can thus realize the function of eachunit of the information processing apparatus of each example embodiment.

The I/O 86 controls input and output of data and a control signal amongthe computer 80 and other input and output apparatuses. The other inputand output apparatuses include, for example, input apparatuses (notillustrated) such as a keyboard, a touch panel, a mouse, and amicrophone connected to the computer 80, output apparatuses such as adisplay, a printer, and a speaker, and an interface among the computer80 and the input and output apparatuses. The I/O 86 may control inputand output of data with other reading or writing apparatuses (notillustrated) for a storage medium.

The communication I/F 87 is a network connection interface performingcommunication between the computer 80 and an external apparatus. Thecommunication I/F 87 may be a network interface for connection to acable line, and may be a network interface for connection to a radioline. For example, the computer 80 realizing the information processingapparatus is connected to at least one camera 10 through a network byusing the communication I/F 87.

Each constituent element of the information processing apparatus of eachexample embodiment is realized by any combination of hardware andsoftware of the computer 80 in FIG. 13. It is understood by a personskilled in the art that there are various modification examples in arealization method and a realization apparatus. The functional blockdiagram illustrating the information processing apparatus of each of theabove-described example embodiments indicates a block in the logicalfunctional unit instead of a configuration in the hardware unit.

The information processing apparatus may be configured with a pluralityof computers 80, and may be realized by a virtual server. The computer80 may be provided in a surveillance target location (for example, in afacility) using the present system, and may be provided in the form ofcloud computing. A camera in a facility may be connected to a network,and may transmit a captured image to the computer 80 configuring aserver on a cloud. The computer 80 in a facility and the computer 80 ona cloud may be combined with each other, and each function of the unitof the information processing apparatus may be distributed to both ofthe computers so as to be executed.

The computer 80 performing an image analysis process may be selectedfrom the computer 80 in a facility and the computer 80 on a clouddepending on situations. For example, there may be a configuration inwhich an authentication process is performed by the computer 80 atnormal times, and an authentication process is performed by the computer80 on the cloud in a case where a highly accurate analysis process isdesired to be performed.

As mentioned above, the example embodiments of this disclosure have beendescribed with reference to the drawings, but these are examples of thisdisclosure, and various configurations other than the description may beemployed. For example, the information processing apparatus further mayinclude a specific person feature information database storing featureinformation of a specific person, and the collation unit 102 may performa collation process on a person for whom a match is not indicated in acollation process using face feature information stored in the firststorage unit 111, by using the specific person feature informationdatabase.

For example, the first storage unit 111 may store face featureinformation of an employee, and the specific person feature informationdatabase may store face feature information of a specific person who isdesired to be specially found, such as a very important person (VIP), ora person on a blacklist as a person with a criminal record or a markedperson. After authentication is performed on employees, a collationprocess with specific person feature information is performed on aperson who is not authenticated, and thus a specific person may bedetected. In a case where a specific person is detected, the output unit120 may output information regarding the specific person.

In a case where there are matches of a predetermined number or more in acollation process with the first registered feature information (faceregistered feature information) or the second registered featureinformation (registered person region feature information), thecollation unit 102 may regard that a collation result indicates a match.The predetermined number may be set on feature information basis. Theinformation processing apparatus 100 may display a menu screen forreceiving setting of the predetermined number, and may set thepredetermined number by receiving an operation from an operator.

EXAMPLES Example 1

In the present example, a description will be made of an example offinding a person which is not authenticated in a predetermined area fora predetermined time or more or by a predetermined time point. Thepresent example is an example using the configuration of the informationprocessing apparatus 100 of the first example embodiment.

FIG. 14 is a diagram illustrating an example of a screen in which animage from the camera 10 imaging a predetermined area is displayed on adisplay (not illustrated) of the computer 80. FIG. 15 is a diagramillustrating a change in data stored in the second storage unit 112. Itshould be noted that FIG. 15 schematically illustrates data contents,and, for example, person region feature information corresponding to aperson region of the person C is written as “C” in the item of “persondetection”.

FIG. 14A illustrates an image screen at a first time point (9:10). It isassumed that person regions of three persons such as the person A, theperson B, and the person C are detected by the collation unit 102. It isassumed that, with respect to the person A and the person B, matchingface feature information is found, and thus authentication thereof iscompleted by the collation unit 102 through a collation process withface feature information stored in the first storage unit 111, but, withrespect to the person C, matching face feature information is not found,and thus authentication thereof is not completed.

An example of information stored in the second storage unit 112 at thistime is illustrated in FIG. 15A. In the example illustrated in FIG. 15A,the second storage unit 112 stores person region feature information ofa person region of which a face is not authenticated, and capturingdate-and-time information of an image from which the person regionfeature information is extracted. Herein, person region featureinformation of the person C of which the face is not authenticated isstored.

Next, FIG. 14B illustrates an image screen at a second time point(9:20), later than the first time point. With respect to the person Aand the person B on whom a collation process is completed by thecollation unit 102 and who are thus authenticated, the “OK” mark isdisplayed in an overlapping manner in association with the personregions on the screen by the output unit 120. With respect to the personC and a person D who cannot be authenticated, the “NG” mark is displayedin an overlapping manner in association with the person regions on thescreen by the output unit 120.

The second storage unit 112 in FIG. 15B further stores person regionfeature information (D) of a person region of the newly detected personD.

FIG. 14C illustrates an image screen at a third time point (9:25), laterthan the second time point. Since the face of the person D has beenauthenticated by the collation unit 102, with respect to the person D,the “OK” mark is displayed in an overlapping manner in association withthe person region on the screen by the output unit 120. Since a face ofthe person C cannot be authenticated by the collation unit 102, withrespect to the person C, the “NG” mark is displayed in an overlappingmanner in association with the person region on the screen by the outputunit 120. Since the face of the person D has been authenticated, asillustrated in FIG. 15C, the record of the person region featureinformation corresponding to the person region of the person D isdeleted from the second storage unit 112 by the deletion unit (notillustrated).

Since only the person region feature information of the person D remainsin the second storage unit 112 at a fourth time point (10:00) after apredetermined time elapses, as illustrated in FIG. 14D, the output unit120 displays an image in which the person C as an unidentified person isclosed up on the display.

FIG. 16 is a diagram illustrating an example of another data structureof the second storage unit 112.

The example illustrated in FIG. 16 is an example using the configurationof the information processing apparatus 100 of the fourth exampleembodiment.

The second storage unit 112 in FIG. 16 stores person region featureinformation, capturing date-and-time information of an image from whichthe person region feature information is extracted, a result of faceauthentication in a person region corresponding to the person regionfeature information (in a case where a face has been authenticated,“completed”, and, in a case where a face cannot be authenticated,“uncompleted”) , and date-and-time information at which a collationprocess is performed in a case where a face is authenticated. It shouldbe noted that FIG. 16 schematically illustrates “completed” and“uncompleted” for face authentication, but, actually, a flag may be set.

In the example illustrated in FIG. 16, at the fourth time point (10:00)after a predetermined time elapses, information regarding person regionfeature information (C) of the person C for which a result of faceauthentication indicates “uncompleted” in the second storage unit 112 isoutput. Herein, as illustrated in FIG. 14D, the output unit 120 displaysan image in which the person C as an unidentified person is closed up onthe display.

Example 2

In the present example, a description will be made of an example offinding a person who cannot be authenticated at least from entry into apath through an entrance to exit from the path through an exit by usinga plurality of cameras provided along the path. The present example isan example using the configuration of the information processingapparatus 100 of the sixth example embodiment.

FIG. 17 is a diagram schematically illustrating a situation of a placewhere four cameras (10-1 to 10-4) are provided along a certain passage.The camera 10-1 and the camera 10-4 are provided at both ends of thepath. In the middle of the passage, the camera 10-2 is provided nearerto the camera 10-1, and the camera 10-3 is provided nearer to the camera10-4.

A plurality of persons A to F are walking along the passage. An arrowadded to each person indicates an advancing direction of the person. Itis assumed that gates (not illustrated) through which a person entersand exits the passage are provided at the positions of the camera 10-1and the camera 10-4.

States after times elapse are illustrated in FIGS. 17A, 17B, and 17C inthis order. For example, the person D enters the passage through theentrance at the time point in FIG. 17A, and is passing in front of thecamera 10-2 at the time point in FIG. 17B. The person D is passing infront of the camera 10-3 at the time point in FIG. 17C. Faceauthentication of the person D is completed immediately after the personD enters the passage.

In FIG. 17, a person with the “OK” mark indicates a person of which faceauthentication is completed, and a person with the “NG” mark indicates aperson of which a face cannot be authenticated. As in FIG. 17, theoutput unit 120 may combine images captured by the respective cameraswith each other, and may display a mark indicating a result of faceauthentication in an overlapping manner. Alternatively, a markindicating a result of face authentication may be displayed to overlapan image from each camera.

FIG. 18 is a diagram illustrating a change in data stored in the secondstorage unit 112. In the present example embodiment, the second storageunit 112 includes a plurality of tables. The second storage unitincludes four tables T1 to T4 in which person region feature informationdetected in an image captured by each camera and detection date-and-timeinformation are stored, and a table T5 in which pieces of person regionfeature information detected by the respective cameras are integratedwith each other, and a face authentication result in the collation unit102 and the authentication date and time are stored for each personregion of the person region feature information.

States in which times elapse are illustrated in FIGS. 18A, 18B, and 18Cin this order.

For example, the person B is passing in front of the camera 10-3 at thetime point in FIG. 18A, and person region feature information isdetected in images from the camera 10-4 and the camera 10-3 so as to bestored in the table T4 and the table T3. Information indicating that aface cannot be authenticated by the collation unit 102 is stored in thetable T5.

The person B is passing in front of the camera 10-2 at the time point inFIG. 18B, and person region feature information is also detected in animage from the camera 10-2 so as to be stored in the table T2.Information indicating that the face cannot be authenticated by thecollation unit 102 is stored in the table T5.

For example, the person B is passing in front of the camera 10-1 at thetime point in FIG. 18C, and person region feature information isdetected in an image from the camera 10-1 so as to be stored in thetable T1. Information indicating that the face cannot be authenticatedby the collation unit 102 is stored in the table T5. Therefore, the faceof the person B is not authenticated in the image captured by the lastcamera of the path, and thus the output unit 120 outputs an alert forthe person B.

Example 3

In the present example, an image captured by a surveillance camera in anairport is used. The registration unit 104 stores person region featureinformation of a person in the airport in the second storage unit 112.

The camera acquires passport information and images a face of a personduring check-in, and the registration unit 104 stores face featureinformation of the person who has checked in in the first storage unit111 of the storage unit 110 in association with the passportinformation.

The collation unit 102 performs a face authentication process whiletracking a person by using registered person region feature informationin an image captured by each camera in the airport. A person for whomcollation does not succeed for a reference time or more is specified,and the output unit 120 outputs an alert. Here, the person for whomcollation does not succeed for a reference time or more is a markedperson who does not check in and is hanging around the airport.

Among persons who have checked in and of which face feature informationis stored in the first storage unit 111 of the storage unit 110, aperson having passed security inspection is excluded from a collationtarget. Specifically, face data is deleted from the first storage unit111. Alternatively, a flag is added to face feature information in thefirst storage unit 111, and the person is excluded from a collationtarget.

As mentioned above, this disclosure has been described with reference tothe example embodiments and the Examples, but this disclosure is notlimited to the example embodiments and Examples. The configuration ordetails of this disclosure may be subjected to various changes which canbe understood by a person skilled in the art within the scope of thisdisclosure.

It should be noted that acquisition and use of information regarding auser in this disclosure are assumed to be performed legally.

Some or all of the above-described example embodiments may be disclosedas in the following appendix, but are not limited thereto.

1. An information processing apparatus including:

a collation unit that collates first feature information extracted froma person included in a first image with first registered featureinformation stored in a storage unit; and

a registration unit that, in a case where the first feature informationis unable to be extracted from the person or a collation result in thecollation unit indicates a mismatch, stores second feature informationextracted from the person in the storage unit as second registeredfeature information,

in which the collation unit collates second feature informationextracted from a person included in a second image with the secondregistered feature information stored in the storage unit, and thusspecifies the person corresponding to the second registered featureinformation in the second image.

2. The information processing apparatus according to 1,

in which the collation unit collates the first feature informationextracted from the person specified in the second image with the firstregistered feature information stored in the storage unit.

3. An information processing apparatus including:

a collation unit that collates first feature information extracted froma person included in a first image with first registered featureinformation stored in a storage unit; and

a registration unit that stores second feature information extractedfrom the person in the storage unit as second registered featureinformation,

in which the collation unit collates second feature informationextracted from a person included in a second image with the secondregistered feature information stored in the storage unit, and thusspecifies the person corresponding to the second registered featureinformation in the second image, and

in which, in a case where the first feature information is unable to beextracted from the person included in the first image, or a collationresult between the first feature information extracted from the personincluded in the first image and the first registered feature informationindicates a mismatch, the collation unit collates first featureinformation extracted from the person specified in the second image withthe first registered feature information stored in the storage unit.

4. The information processing apparatus according to any one of 1. to3.,

in which the collation unit repeatedly performs a process of specifyinga person corresponding to the second registered feature information inthe second image and a process of collating the first featureinformation extracted from the specified person with the firstregistered feature information until the collation result indicates amatch.

5. The information processing apparatus according to any one of 1. to4., further including:

a deletion unit that, in a case where a collation result between thefirst feature information extracted from the person and the firstregistered feature information stored in the storage unit indicates amatch, deletes the second registered feature information of the personspecified in the second image from the storage unit.

6. The information processing apparatus according to any one of 1. to5.,

in which the second image is captured by a camera which is differentfrom a camera capturing the first image.

7. The information processing apparatus according to any one of 1. to6.,

in which the first image is captured by a first camera among a pluralityof cameras provided along a path, and the second image is captured by asecond camera located after the first camera in a direction along thepath.

8. The information processing apparatus according to 7., furtherincluding:

an output unit that, in a case where a collation result with the firstregistered feature information indicates a mismatch with respect to theperson specified by the collation unit in the second image generated bythe second camera provided last or at a predetermined position in thedirection along the path among a plurality of the second cameras,outputs information regarding the second feature information extractedfrom the person.

9. The information processing apparatus according to any one of 1. to8.,

in which the first image is captured at a first timing, and the secondimage is captured after the first timing.

10. The information processing apparatus according to any one of 1. to9.,

in which the registration unit stores information indicating thecollation result between the first feature information extracted fromthe person included in the first image and the first registered featureinformation in the storage unit in association with the secondregistered feature information, and

in which, in a case where the information indicating the collationresult indicates a mismatch as the collation result, the collation unitcollates the first feature information extracted from the personspecified in the second image with the first registered featureinformation stored in the storage unit.

11. The information processing apparatus according to any one of 1. to10.,

in which the registration unit stores information indicating that thefirst feature information is unable to be extracted from the personincluded in the first image, in the storage unit in association with thesecond registered feature information, and

in which, in a case where the information indicates that the firstfeature information is unable to be extracted from the person includedin the first image, the collation unit collates the first featureinformation extracted from the person specified in the second image withthe first registered feature information stored in the storage unit.

12. The information processing apparatus according to any one of 1. to11.,

in which the registration unit stores the second feature information inthe storage unit along with time information,

the information processing apparatus further including an output unitthat, in a case where the first feature information is unable to beextracted from the person specified in the second image or the collationresult with the first feature information extracted from the personspecified in the second image indicates a mismatch even though areference time or more elapses from a time point indicated by the timeinformation, outputs information regarding the second featureinformation.

13. The information processing apparatus according to any one of 1. to12.,

in which, in a case where matches of a predetermined number or moreoccur in a collation process with the first registered featureinformation or the second registered feature information, the collationunit regards that the collation result indicates a match.

14. The information processing apparatus according to any one of 1. to13.,

in which the first feature information is face feature information, andthe second feature information is feature information including a regionother than a face.

15. The information processing apparatus according to any one of 1. to14., further including:

an output unit that outputs information regarding the second registeredfeature information which is stored in the storage unit at a referencetiming.

16. An information processing system including:

a collation unit that collates first feature information extracted froma person included in a first image with first registered featureinformation stored in a storage unit; and

a registration unit that, in a case where the first feature informationis unable to be extracted from the person or a collation result in thecollation unit indicates a mismatch, stores second feature informationextracted from the person in the storage unit as second registeredfeature information,

in which the collation unit collates second feature informationextracted from a person included in a second image with the secondregistered feature information stored in the storage unit, and thusspecifies the person corresponding to the second registered featureinformation in the second image.

17. The information processing system according to 16.,

in which the collation unit collates the first feature informationextracted from the person specified in the second image with the firstregistered feature information stored in the storage unit.

18. An information processing system including:

a collation unit that collates first feature information extracted froma person included in a first image with first registered featureinformation stored in a storage unit; and

a registration unit that stores second feature information extractedfrom the person in the storage unit as second registered featureinformation,

in which the collation unit collates second feature informationextracted from a person included in a second image with the secondregistered feature information stored in the storage unit, and thusspecifies the person corresponding to the second registered featureinformation in the second image, and

in which, in a case where the first feature information is unable to beextracted from the person included in the first image, or a collationresult between the first feature information extracted from the personincluded in the first image and the first registered feature informationindicates a mismatch, the collation unit collates first featureinformation extracted from the person specified in the second image withthe first registered feature information stored in the storage unit.

19. The information processing system according to any one of 16. to18.,

in which the collation unit repeatedly performs a process of specifyinga person corresponding to the second registered feature information inthe second image and a process of collating the first featureinformation extracted from the specified person with the firstregistered feature information until the collation result indicates amatch.

20. The information processing system according to any one of 16. to19., further including:

a deletion unit that, in a case where a collation result between thefirst feature information extracted from the person specified in thesecond image and the first registered feature information stored in thestorage unit indicates a match, deletes the second registered featureinformation of the person from the storage unit.

21. The information processing system according to any one of 16. to20.,

in which the second image is captured by a camera which is differentfrom a camera capturing the first image.

22. The information processing system according to any one of 16. to21.,

in which the first image is captured by a first camera among a pluralityof cameras provided along a path, and the second image is captured by asecond camera located after the first camera in a direction along thepath.

23. The information processing system according to 22., furtherincluding:

an output unit that, in a case where a collation result with the firstregistered feature information indicates a mismatch with respect to theperson specified by the collation unit in the second image generated bythe second camera provided last or at a predetermined position in thedirection along the path among a plurality of the second cameras,outputs information regarding the second feature information extractedfrom the person.

24. The information processing system according to any one of 16. to23.,

in which the first image is captured at a first timing, and the secondimage is captured after the first timing.

25. The information processing system according to any one of 16. to24.,

in which the registration unit stores information indicating thecollation result between the first feature information extracted fromthe person included in the first image and the first registered featureinformation in the storage unit in association with the secondregistered feature information, and

in which, in a case where the information indicating the collationresult indicates a mismatch as the collation result, the collation unitcollates the first feature information extracted from the personspecified in the second image with the first registered featureinformation stored in the storage unit.

26. The information processing system according to any one of 16. to25.,

in which the registration unit stores information indicating that thefirst feature information is unable to be extracted from the personincluded in the first image, in the storage unit in association with thesecond registered feature information, and

in which, in a case where the information indicates that the firstfeature information is unable to be extracted from the person includedin the first image, the collation unit collates the first featureinformation extracted from the person specified in the second image withthe first registered feature information stored in the storage unit.

27. The information processing system according to any one of 16. to26.,

in which the registration unit stores the second feature information inthe storage unit along with time information,

the information processing system further including an output unit that,in a case where the first feature information is unable to be extractedfrom the person specified in the second image or the collation resultwith the first feature information extracted from the person specifiedin the second image indicates a mismatch even though a reference time ormore elapses from a time point indicated by the time information,outputs information regarding the second feature information.

28. The information processing system according to any one of 16. to27.,

in which, in a case where matches of a predetermined number or moreoccur in a collation process with the first registered featureinformation or the second registered feature information, the collationunit regards that the collation result indicates a match.

29. The information processing system according to any one of 16. to28.,

in which the first feature information is face feature information, andthe second feature information is feature information including a regionother than a face.

30. The information processing system according to any one of 16. to29., further including:

an output unit that outputs information regarding the second registeredfeature information which is stored in the storage unit at a referencetiming.

31. An information processing method executed by an informationprocessing apparatus, the method including:

collating first feature information extracted from a person included ina first image with first registered feature information stored in astorage unit;

in the storage unit as second registered feature information in a casewhere the first feature information is unable to be extracted from theperson or a collation result indicates a mismatch, storing secondfeature information extracted from the person; and

collating second feature information extracted from a person included ina second image with the second registered feature information stored inthe storage unit, and thus specifying the person corresponding to thesecond registered feature information in the second image.

32. The information processing method executed by an informationprocessing apparatus according to 31., the method comprising:

collating the first feature information extracted from the personspecified in the second image with the first registered featureinformation stored in the storage unit.

33. An information processing method executed by an informationprocessing apparatus, the method including:

collating first feature information extracted from a person included ina first image with first registered feature information stored in astorage unit;

storing second feature information extracted from the person in thestorage unit as second registered feature information;

collating second feature information extracted from a person included ina second image with the second registered feature information stored inthe storage unit, and thus specifying the person corresponding to thesecond registered feature information in the second image; and

in a case where the first feature information is unable to be extractedfrom the person included in the first image, or a collation resultbetween the first feature information extracted from the person includedin the first image and the first registered feature informationindicates a mismatch, collating first feature information extracted fromthe person specified in the second image with the first registeredfeature information stored in the storage unit.

34. The information processing method executed by an informationprocessing apparatus according to any one of 31. to 33., the methodcomprising:

repeatedly performing a process of specifying a person corresponding tothe second registered feature information in the second image and aprocess of collating the first feature information extracted from thespecified person with the first registered feature information until thecollation result indicates a match.

35. The information processing method executed by an informationprocessing apparatus according to any one of 31. to 34., the methodfurther including:

in a case where a collation result between the first feature informationextracted from the person specified in the second image and the firstregistered feature information stored in the storage unit indicates amatch, deleting the second registered feature information of the personfrom the storage unit.

36. The information processing method according to any one of 31. to35.,

in which the second image is captured by a camera which is differentfrom a camera capturing the first image.

37. The information processing method according to any one of 31. to36.,

in which the first image is captured by a first camera among a pluralityof cameras provided along a path, and the second image is captured by asecond camera located after the first camera in a direction along thepath.

38. The information processing method executed by an informationprocessing apparatus according to 37., the method comprising:

in a case where a collation result with the first registered featureinformation indicates a mismatch with respect to the person specified inthe second image generated by the second camera provided last or at apredetermined position in the direction along the path among a pluralityof the second cameras, outputting information regarding the secondfeature information extracted from the person.

39. The information processing method according to any one of 31. to38.,

in which the first image is captured at a first timing, and the secondimage is captured after the first timing.

40. The information processing method executed by an informationprocessing apparatus according to any one of 31. to 39., the methodcomprising:

storing information indicating the collation result between the firstfeature information extracted from the person included in the firstimage and the first registered feature information in the storage unitin association with the second registered feature information; and

collating the first feature information extracted from the personspecified in the second image with the first registered featureinformation stored in the storage unit in a case where the informationindicating the collation result indicates a mismatch as the collationresult.

41. The information processing method executed by an informationprocessing apparatus according to any one of 31. to 40., the methodcomprising:

storing information indicating that the first feature information isunable to be extracted from the person included in the first image, inthe storage unit in association with the second registered featureinformation; and

collating the first feature information extracted from the personspecified in the second image with the first registered featureinformation stored in the storage unit in a case where the informationindicates that the first feature information is unable to be extractedfrom the person included in the first image.

42. The information processing method executed by an informationprocessing apparatus according to any one of 31. to 41., the methodfurther including:

storing the second feature information in the storage unit along withtime information; and

in a case where the first feature information is unable to be extractedfrom the person specified in the second image or the collation resultwith the first feature information extracted from the person specifiedin the second image indicates a mismatch even though a reference time ormore elapses from a time point indicated by the time information,outputting information regarding the second feature information.

43. The information processing method executed by an informationprocessing apparatus according to any one of 31. to 42., the methodcomprising:

in a case where matches of a predetermined number or more occur in acollation process with the first registered feature information or thesecond registered feature information, regarding that the collationresult indicates a match.

44. The information processing method according to any one of 31. to43.,

in which the first feature information is face feature information, andthe second feature information is feature information including a regionother than a face.

45. The information processing method executed by an informationprocessing apparatus according to any one of 31. to 44., the methodfurther including:

outputting information regarding the second registered featureinformation which is stored in the storage unit at a reference timing.

46. A program causing a computer to execute:

a procedure of collating first feature information extracted from aperson included in a first image with first registered featureinformation stored in a storage unit;

a procedure of storing, in a case where the first feature information isunable to be extracted from the person or a collation result in thecollation procedure indicates a mismatch, second feature informationextracted from the person in the storage unit as second registeredfeature information; and

a procedure of collating second feature information extracted from aperson included in a second image with the second registered featureinformation stored in the storage unit, and thus specifying the personcorresponding to the second registered feature information in the secondimage.

47. The program according to 46., causing the computer to execute:

a procedure of collating the first feature information extracted fromthe person specified in the second image with the first registeredfeature information stored in the storage unit.

48. A program causing a computer to execute:

a procedure of collating first feature information extracted from aperson included in a first image with first registered featureinformation stored in a storage unit;

a procedure of storing second feature information extracted from theperson in the storage unit as second registered feature information;

a procedure of collating second feature information extracted from aperson included in a second image with the second registered featureinformation stored in the storage unit, and thus specifying a personcorresponding to the second registered feature information in the secondimage; and

a procedure of collating, in a case where the first feature informationis unable to be extracted from the person included in the first image,or a collation result between the first feature information extractedfrom the person included in the first image and the first registeredfeature information indicates a mismatch, first feature informationextracted from the person specified in the second image with the firstregistered feature information stored in the storage unit.

49. The program according to any one of 46. to 48.,

a procedure of repeatedly performing a process of specifying a personcorresponding to the second registered feature information in the secondimage and a process of collating the first feature information extractedfrom the specified person with the first registered feature informationuntil the collation result indicates a match.

50. The program according to any one of 46. to 49., causing the computerto further execute:

a procedure of deleting, in a case where a collation result between thefirst feature information extracted from the person specified in thesecond image and the first registered feature information stored in thestorage unit indicates a match, the second registered featureinformation of the person from the storage unit.

51. The program according to any one of 46. to 50.,

in which the second image is captured by a camera which is differentfrom a camera capturing the first image.

52. The program according to any one of 46. to 51.,

in which the first image is captured by a first camera among a pluralityof cameras provided along a path, and the second image is captured by asecond camera located after the first camera in a direction along thepath.

53. The program according to 52., causing the computer to execute:

a procedure of outputting, in a case where a collation result with thefirst registered feature information indicates a mismatch with respectto the person specified in the second image generated by the secondcamera provided last or at a predetermined position in the directionalong the path among a plurality of the second cameras, informationregarding the second feature information extracted from the person.

54. The program according to any one of 46. to 53.,

in which the first image is captured at a first timing, and the secondimage is captured after the first timing.

55. The program according to any one of 46. to 54., causing the computerto execute:

a procedure of storing information indicating the collation resultbetween the first feature information extracted from the person includedin the first image and the first registered feature information in thestorage unit in association with the second registered featureinformation, and a procedure of collating the first feature informationextracted from the person specified in the second image with the firstregistered feature information stored in the storage unit in a casewhere the information indicating the collation result indicates amismatch as the collation result.

56. The program according to any one of 46. to 55., causing the computerto execute:

a procedure of storing information indicating that the first featureinformation is unable to be extracted from the person included in thefirst image, in the storage unit in association with the secondregistered feature information, and

a procedure of collating the first feature information extracted fromthe person specified in the second image with the first registeredfeature information stored in the storage unit in a case where theinformation indicates that the first feature information is unable to beextracted from the person included in the first image.

57. The program according to any one of 46. to 56., causing the computerto further execute:

a procedure of storing the second feature information in the storageunit along with time information; and

a procedure of outputting, in a case where the first feature informationis unable to be extracted from the person specified in the second imageor the collation result with the first feature information extractedfrom the person specified in the second image indicates a mismatch eventhough a reference time or more elapses from a time point indicated bythe time information, information regarding the second featureinformation.

58. The program according to any one of 46. to 57.,

in which, in a case where matches of a predetermined number or moreoccur in a collation process with the first registered featureinformation or the second registered feature information, the computerexecutes a procedure of regarding that the collation result indicates amatch.

59. The program according to any one of 46. to 58.,

in which the first feature information is face feature information, andthe second feature information is feature information including a regionother than a face.

60. The program according to any one of 46. to 59., causing the computerto further execute:

a procedure of outputting information regarding the second registeredfeature information which is stored in the storage unit at a referencetiming.

61. An information processing apparatus processing an image generated byeach of a plurality of cameras provided along a path, comprising:

a collation unit that collates first feature information extracted froma person included in a first image captured by a first camera among theplurality of cameras, with first registered feature information storedin a storage unit; and

a registration unit that stores second feature information extractedfrom the person in the storage unit as second registered featureinformation in a case where the first feature information is unable tobe extracted from the person or a collation result in the collation unitindicates a mismatch,

in which the collation unit collates the second feature informationextracted from a person included in a second image captured by a secondcamera located after the first camera in a direction along the path,with the second registered feature information stored in the storageunit, and thus specifies a person corresponding to the second registeredfeature information in the second image.

62. An information processing apparatus processing an image generated byeach of a plurality of cameras provided along a path, comprising:

a collation unit that collates first feature information extracted froma person included in a first image captured by a first camera among theplurality of cameras, with first registered feature information storedin a storage unit; and

a registration unit that stores second feature information extractedfrom the person in the storage unit,

in which the collation unit collates second feature informationextracted from a person included in a second image captured by a secondcamera located after the first camera in a direction along the path,with second registered feature information stored in the storage unit,and thus specifies the person in the second image, and

in which, in a case where the first feature information is unable to beextracted from the person included in the first image, or a collationresult with the first feature information extracted from the personincluded in the first image indicates a mismatch, the collation unitcollates the first feature information extracted from the person in thesecond image with the first registered feature information stored in thestorage unit.

What is claimed is:
 1. An information processing apparatus comprising: acollation unit that collates first feature information extracted from aperson included in a first image with first registered featureinformation stored in a storage unit; and a registration unit that, in acase where the first feature information is unable to be extracted fromthe person or a collation result in the collation unit indicates amismatch, stores second feature information extracted from the person inthe storage unit as second registered feature information, wherein thecollation unit collates second feature information extracted from aperson included in a second image with the second registered featureinformation stored in the storage unit, and thus specifies the personcorresponding to the second registered feature information in the secondimage.
 2. The information processing apparatus according to claim 1,wherein the collation unit collates the first feature informationextracted from the person specified in the second image with the firstregistered feature information stored in the storage unit.
 3. (canceled)4. The information processing apparatus according to claim 1, whereinthe collation unit repeatedly performs a process of specifying a personcorresponding to the second registered feature information in the secondimage and a process of collating the first feature information extractedfrom the specified person with the first registered feature informationuntil the collation result indicates a match.
 5. The informationprocessing apparatus according to claim 1, further comprising: adeletion unit that, in a case where a collation result between the firstfeature information extracted from the person specified in the secondimage and the first registered feature information stored in the storageunit indicates a match, deletes the second registered featureinformation of the person from the storage unit.
 6. The informationprocessing apparatus according to claim 1, wherein the second image iscaptured by a camera which is different from a camera capturing thefirst image.
 7. The information processing apparatus according to claim1, wherein the first image is captured by a first camera among aplurality of cameras provided along a path, and the second image iscaptured by a second camera located after the first camera in adirection along the path.
 8. The information processing apparatusaccording to claim 7, further comprising: an output unit that, in a casewhere a collation result with the first registered feature informationindicates a mismatch with respect to the person specified by thecollation unit in the second image generated by the second cameraprovided last or at a predetermined position in the direction along thepath among a plurality of the second cameras, outputs informationregarding the second feature information extracted from the person. 9.The information processing apparatus according to claim 1, wherein thefirst image is captured at a first timing, and the second image iscaptured after the first timing.
 10. The information processingapparatus according to claim 1, wherein the registration unit storesinformation indicating the collation result between the first featureinformation extracted from the person included in the first image andthe first registered feature information in the storage unit inassociation with the second registered feature information, and wherein,in a case where the information indicating the collation resultindicates a mismatch as the collation result, the collation unitcollates the first feature information extracted from the personspecified in the second image with the first registered featureinformation stored in the storage unit.
 11. The information processingapparatus according to claim 1, wherein the registration unit storesinformation indicating that the first feature information is unable tobe extracted from the person included in the first image, in the storageunit in association with the second registered feature information, andwherein, in a case where the information indicates that the firstfeature information is unable to be extracted from the person includedin the first image, the collation unit collates the first featureinformation extracted from the person specified in the second image withthe first registered feature information stored in the storage unit. 12.The information processing apparatus according to claim 1, wherein theregistration unit stores the second feature information in the storageunit along with time information, the information processing apparatusfurther comprising an output unit that, in a case where the firstfeature information is unable to be extracted from the person specifiedin the second image or the collation result with the first featureinformation extracted from the person specified in the second imageindicates a mismatch even though a reference time or more elapses from atime point indicated by the time information, outputs informationregarding the second feature information.
 13. The information processingapparatus according to claim 1, wherein, in a case where matches of apredetermined number or more occur in a collation process with the firstregistered feature information or the second registered featureinformation, the collation unit regards that the collation resultindicates a match.
 14. The information processing apparatus according toclaim 1, wherein the first feature information is face featureinformation, and the second feature information is feature informationincluding a region other than a face.
 15. The information processingapparatus according to claim 1, further comprising: an output unit thatoutputs information regarding the second registered feature informationwhich is stored in the storage unit at a reference timing. 16-30.(canceled)
 31. An information processing method executed by aninformation processing apparatus, the method comprising: collating firstfeature information extracted from a person included in a first imagewith first registered feature information stored in a storage unit; in acase where the first feature information is unable to be extracted fromthe person or a collation result indicates a mismatch, storing secondfeature information extracted from the person in the storage unit assecond registered feature information; and collating second featureinformation extracted from a person included in a second image with thesecond registered feature information stored in the storage unit, andthus specifying the person corresponding to the second registeredfeature information in the second image.
 32. The information processingmethod executed by an information processing apparatus according toclaim 31, the method comprising: collating the first feature informationextracted from the person specified in the second image with the firstregistered feature information stored in the storage unit. 33.(canceled)
 34. The information processing method executed by aninformation processing apparatus according to claim 31, the methodcomprising: repeatedly performing a process of specifying a personcorresponding to the second registered feature information in the secondimage and a process of collating the first feature information extractedfrom the specified person with the first registered feature informationuntil the collation result indicates a match.
 35. The informationprocessing method executed by an information processing apparatusaccording to claim 31, the method further comprising: in a case where acollation result between the first feature information extracted fromthe person specified in the second image and the first registeredfeature information stored in the storage unit indicates a match,deleting the second registered feature information of the person fromthe storage unit.
 36. The information processing method according toclaim 31, wherein the second image is captured by a camera which isdifferent from a camera capturing the first image. 37-45. (canceled) 46.A non-transitory computer readable storage medium storing a programcausing a computer to execute: a procedure of collating first featureinformation extracted from a person included in a first image with firstregistered feature information stored in a storage unit; a procedure ofstoring, in a case where the first feature information is unable to beextracted from the person or a collation result in the collationprocedure indicates a mismatch, second feature information extractedfrom the person in the storage unit as second registered featureinformation; and a procedure of collating second feature informationextracted from a person included in a second image with the secondregistered feature information stored in the storage unit, and thusspecifying the person corresponding to the second registered featureinformation in the second image. 47-60. (canceled)